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Abstract 

This paper presents an overview of the 
proposed Lyndon B. Johnson Space Center (JSC) 
precompetitive, dual-use technology investment 
project in robotics. New robotic technology in 
advanced robots, which can recognize and respond 
to their environments and to spoken human 
supervision so as to perform a variety of combined 
mobility and manipulation tasks in various sectors, 
is an objective of this work. Inthe U S. economy, 
such robots offer the benefits of improved global 
competitiveness in a critical industrial sector; 
improved productivity by the end users of these 
robots; a growing robotics industry that produces 
jobs and profits; lower cost health care delivery 
with quality improvements; and, as these 
"intelligent" robots become acceptable through- 
out society, an increase in the standard of living for 
everyone. In space, such robots will provide 
improved safety, reliability, and productivity as 
Space Station evolves, and will enable human 
space exploration (by human/robot teams). 

The proposed effort consists of partnerships 
between manufacturers, universities, and JSC to 
develop working production prototypes of these 
robots by leveraging current development by both 
sides. Currently targeted applications are in the 
manufacturing, health care, services, and 
construction sectors of the U.S. economy and in the 
inspection, servicing, maintenance, and repair 
aspects of space exploration. But the focus is on 
the generic software architecture and standard- 
ized interfaces for custom modules tailored for the 
various applications allowing end users to 
customize a robot as PC users customize PC's. 
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Production prototypes would be completed in 
5 years under this proposal. 


1. Introduction 

This paper suggests a large number of 
opportunities for robotic manufacturers, integra- 
tors, potential buyers/users of robots, commercial 
technology developers, and universities to work 
with the NASA JSC Automation and Robotics 
Division, with NASA funding a major portion of the 
development. The focus is intelligent robotics as 
partial solutions to productivity problems in 
several sectors of application. The stage of devel- 
opment addressed is precommercial. In each case 
dual use is a prerequisite: there must be a space 
use as well as a nonspace, commercial use. 
Generally, this is easily the case. 

The specific motivation and rationale for this 
NASA JSC proposed technology investment pro- 
gram is detailed in Erickson 1 . The general policy 
that sets the context for the NASA technology 
investment program, which will begin in 1994, is 
given in Clinton and Gore 2 . 

It is important to understand that although a 
set of objectives, an approach, and a number of 
tasks are suggested here, these are meant to 
stimulate the creative thought process of those in 
nonaerospace and aerospace industry to propose 
objectives, approaches, and tasks that they believe, 
due to their involvement with their commercial 
buyers/users, would be economic and profitable as 
a result of jointly funded developmental efforts 
with NASA. 

Intelligent robotics is the use of robotic 
systems in solving problems in tasks and environ- 
ments where the robot's ability to acquire and 
apply knowledge and skills to achieve stated goals 
in the face of variations, difficulties, and complex- 
ities imposed by a dynamic environment having 
significant unpredictability is crucial to success. 

This means the robots can recognize and respond 
to their environments at the pace of their environ- 
ments and to spoken human supervision so as to 
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perform a variety of mobility and manipulation 
tasks. This does not require broad-based general 
intelligence or common sense by the robot. 

These robots are capable of significant, 
autonomous reaction to unpredictable events, yet 
are subject to optional human supervision during 
operation in a natural way, such as by voice. We 
refer to this capability in the supervised robot as 
"adjustable autonomy." Also, a key essence is that 
previously acquired knowledge is combined with 
knowledge acquired at the instant of task 
performance. 

The overall approach can be summarized as 
capitalizing on a software architecture that can be 
viewed as generic and modular, and hardware 
approaches that are modular, reconfigurable, and 
extendible. Many of the software modules, such as 
a deliberative planner, world model, and natural 
language interface, can also be viewed as generic. 
Other software is bundled with certain hardware; 
e.g., sensing software is bundled with specific 
sensor hardware. This leads to the concept of a 
modular, end-user customized robot, put together 
from modules with standard interfaces 3 - 4 - 5 such 
as users do with a personal computer. An integra- 
ted computer aided concurrent engineering 
environment that we are working on 6 is a way to 
achieve close teamwork by geographically distrib- 
uted "virtual" teams to develop the production 
prototypes. 

JSC can be a key partner in this dual-use 
technology investment program in intelligent 
robotics for two reasons: (1) human space explor- 
ation missions require supervised intelligent 
robotics as enabling tools 7 - 8 and, hence, must 
develop or have developed supervised intelligent 
robotic systems and (2) intelligent robotic technol- 
ogy is being developed for space applications at 
JSC (but has a strong crosscutting or generic flavor) 
that is advanci ng the state of the art 9 - 1 ° and is 
producing both skilled personnel and adaptable 
developmental infrastructure, such as low cost 
simulation environments for software testing and 
integrated testbeds for complete prototype 
testing. JSC also has a Small Business Innovative 
Research (SB I R) program 11 for intelligent robotics, 
which is underutilized and has no commercial cost 
sharing requirement. It is limited in scope to about 
$0.6 million and 2 years in Phase II efforts. 

A key element in the cutting edge intelligent 
robotics technology work at JSC is an understand- 
ing of and solution approach to the key issue of 
melding artificial intelligence planners with 


reactive capabilities. Artificial intelligence 
planners offer goal-achieving planning, but also 
high-time variance due to searching. Reactive 
capabilities are needed to deal safely in real time 
with dynamic, unpredictable environments at the 
pace of the dynamics 9 . A second key element that 
JSC brings is an approach to improved robotic reli- 
ability as required for space, but also useful in 
industry. A third key element that JSC brings to 
cutting edge technology is an understanding of 
and solution approach to the key issue of robotic 
safety while maintaining productivity. 

Of all these elements, the personnel skilled in 
the state of the art and knowledgeable about the 
technology are the most important. 


2. Overview of Proposed Activities 

New robotic technology in advanced robots 
that can recognize and respond to their environ- 
ments and to spoken human supervision so as to 
perform a variety of mobility and manipulation 
tasks in various sectors is an objective of this 
proposed effort. In the U.S. economy, such robots 
offer the benefits of improved global competitive- 
ness in a critical industrial sector; improved 
productivity by the end users of these robots; a 
growing robotics industry that produces jobs and 
profits; lower cost health care delivery with quality 
improvements; and, as these "intelligent" robots 
become acceptable throughout society, an 
increase in the standard of living for everyone 12 . 

In space, such robots will provide improved safety, 
reliability, and productivity as Space Station 
evolves, and will enable human space exploration 
(by human/robot teams). 

The proposed effort consists of partnerships 
between manufacturers, users, universities, and 
JSC to develop working production prototypes of 
these robots by leveraging current development by 
manufacturers and JSC. Currently targeted appli- 
cations are in the manufacturing, health care, 
services, and construction sectors of the U.S. 
economy and in the inspection, servicing, mainte- 
nance, and repair aspects of space exploration. But 
the focus is on the generic software architecture 
and standardized interfaces for custom modules 
tailored for the various applications, allowing end 
users to customize a robot as personal computer 
users customize PC's. Production prototypes would 
be completed in 5 years under this proposal, as 
would automated developmental environments 
and integrated testbeds. 
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JSC possesses the required core skills in its civil 
service and contractors to form the nucleus of the 
multiple partnerships. Current technology integra- 
tion efforts at JSC include the EVA helper/retriever 
supervised intelligent robot 10 ( the mobile robotics 
testbed project, and the Soda-Pup entry in the 
AAAI national robotics competition (1992 award 
winner). In addition, JSC is responsible for 
engineering upgrades to the Shuttle Remote 
Manipulator Systems, integration of the Mobile 
Servicing Systems and Special Purpose Dexterous 
Manipulator into Space Station, and numerous 
robotics technology efforts. 

User coordination involves interested 
manufacturers with deployed robots. Joint facility 
sharing and temporary personnel exchange are 
possible. 

The overall set of activities has been grouped 
into the following seven related categories of 
tasks, each with its own objectives and approach. 

1. Problem-Solving Insertion of Robot 
Intelligence Technology 

2. Generic Intelligent Robotics Software 
Architecture 

3. Modular Manipulation and Mobility for 
Robotics 

4. Integrated Sensing and Perception Capabilities 
for Robotics 

5. Robotic Surrogates for Human Grasping and 
Manipulation 

6. Integrated Prototyping Environment for 
Robotics 

7. Robotic Applications in Advanced 
Manufacturing, Health Care, Service Industries, 
and Construction 

The following sections present the objectives, 
approach, and benefits for each of these catego- 
ries of tasks and give the titles of the set of tasks 
grouped into that category. One-page task 
descriptions are available 13 for all tasks, giving 
task objectives, proposed effort, major milestones, 
benefits, and other information. 


Problem-Solving Insertion of Robot Intelligence 
Technology 

The objectives are (1) to work as a team with 
end user industries whose productivity problems 
can be solved by integrating adaptive robots into 
the advanced manufacturing or service process, 


and in so doing to develop a new paradigm of 
product line development for robot manufacturers 
and (2) to provide the robotics industry sensor/ 
software control techniques that will make the 
robots more flexible and attractive for use by end 
user industries. This will impact the end users of 
these robots by improving the end users' efficiency 
and productivity and thus improved global 
competitiveness. This will also stimulate robot 
demand and provide a new way of doing business 
for robot manufacturers. The benefits for space 
will be a healthier robotics industry capable of 
supplying quality robotics at lower costs. 

The proposed effort consists of partnerships 
between manufacturers, integrators, nonprofits, 
and JSC to solve end user problems by integrating 
adaptive robots into end user operations. As part 
of that effort the robot manufacturers' products 
must first be upgraded with sensing and intelligent 
reaction capabilities from new sensor/software 
control technology. A key product is the develop- 
ment, documentation, and refinement of the 
problem-solving insertion process for intelligent 
robotics technology, including end user problem 
identification techniques; problem selection 
criteria; requirements definition; development of 
a solution; integration with the end user people, 
processes, and equipment; user training; and 
continuing user support. 

In related work, JSC is responsible for 
integration of the Mobile Servicing System and 
Special Purpose Dexterous Manipulator into Space 
Station, which gives us the necessary experience 
and insight to help users. 

The eight tasks in this category are the 
following: 

• End User Target Problems Identification 

• End User Problem Selection 

• Selected Problem Requirements Definition 

• Design, Development, Test, and Evaluation of 
Solution 

• Integration with User Equipment, Processes, and 
People 

• User Training Definition 

• Continuing User Support Definition 

• Problem Solving Insertion Process Development, 
Documentation, and Refinement 

The benefits of this problem-solving insertion 
are that the end user businesses obtain a useful 
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solution to their problems. The robot manufactur- 
ers and integrators obtain a better understanding 
of the integration process, not only as part of the 
problem-solving insertion of their products but 
also as part of the requirements for capabilities in 
their products. JSC gets a benefit for space 
applications due to understanding of capabilities 
of intelligent robots required to solve certain types 
of problems. 

Generic Intelligent Robotics Software Architecture 

The objective is a generic, supervised 
intelligent robotics software architecture that 
provides a portable software approach that 
integrates intelligent planning and reactive 
control with sensing and internal representation of 
environment to enable advanced robots that can 
recognize and respond to their surroundings and 
to spoken human supervision in order to perform a 
variety of manipulation and mobility tasks. The 
benefits of such an architecture are the faster 
development time, lower cost, and increased 
adaptable and flexible performance. In turn, these 
provide improved productivity by the end users of 
these robots, whether terrestrial or space. 

The proposed effort consists of partnerships 
between manufacturers, nonprofits, and JSC to 
develop the software and evaluate its charac- 
teristics and robustness in several tasks and 
environments. The design of the software 
architecture, which is the framework (functional 
decomposition) that integrates the separate 
functional modules into a coherent system, is 
dictated in large measure by the tasks and nature 
of the environment. Because both the goal- 
achieving tasks and the partially unpredictable 
nature of the environments are similar on Earth 
and in space, the software architecture can be 
viewed as generic. Many of the software modules, 
such as a deliberative planner, world model 
capability, and natural language interface, can also 
be viewed as generic. Other software is bundled 
with certain hardware; e.g., sensing software is 
bundled with specific sensor hardware. 

Current work on the EVA helper/retriever 
supervised intelligent robot, the mobile robotics 
testbed project, and Soda-Pup project at JSC have 
provided us the necessary insight and experience. 
Not just any architecture will do here. It must solve 
the key issue of combining deliberative goal- 
achieving planning with reactive capabilities in 
such a way as not to I i m it the i ntel I i gence of the 
planner or the safety of the reactive execution. 


The JSC work is believed to offer such a solution. It 
is a practical implementation of the mathematical 
theory of intelligent robots 14 . 

The ten tasks in this category are the 
following: 

• Artificial Intelligence Planning Software 

• Sequencing and Scheduling Software 

• Reactive Controller Software 

• Integration of Natural Language Understanding 
Into Architecture 

• Real-Ti me Speech Planni ng Software 

• World Modeling Software 

• Software Development Environment 

• Integrated Software Architecture 

• Integrated Testing Against Simulated 
Environments 

• Skill Acquisition 

A generic software architecture for super- 
vised intelligent robots will enable portability and 
reuse, major time and cost savings in development 
and testing, more robust and higher quality 
software, and maintenance and training cost 
reductions. People will have a natural means of 
supervision by including task limited natural 
language understanding and speech generation 
software in the robotics software architecture. 
Improved safety of operations is also a benefit. 
These benefits apply in space and in the U S. 
economy. 

Modular Manipulation and Mobility for Robotics 

The objective here is to develop a set of 
standardized modular components that can be 
reconfigured as required into modular robots 
offering a broad spectrum of tasks, reduced system 
costs, reduced weight, reduced mean time to 
repair, changeout of broken components, and 
reduced operator training. As components for an 
integrated prototyping environment for evalu- 
ating alternate approaches to design of robotic 
systems, these contribute to making adaptive 
robots "faster, better, and cheaper." 

The proposed effort consists of partnerships 
with robotic manufacturers, nonprofits, and 
universities to develop working production 
prototypes of a set of standardized modular 
components. The development of standards for 
mechanical and electrical connections and similar 


modular interfaces will be a product as well. Both 
manipulator and mobility systems with robot body 
structures would be developed. Arm sockets, links, 
joints, actuators, and sensors would be designed 
and developed to standards for manipulators. 
Wheels, tracks, suspension, drive train motors, 
gears, brakes, drive control electronics, structure, 
pan/tilt units, power and communications 
subsystems, and sensors would be designed and 
developed to standards for mobility and body 
systems. 

We have a current effort in designing 
modular components for space manipulation 4 . 

The eight tasks in this category are the 
following: 

• Manipulator Socket, Link, Joint, Actuator, and 
Sensor Modular Component Standards 
Development 

• Manipulator Modular Component Designs 

• Manipulator Modular Component Development 

• Mobility Modular Component Standards 
Development 

• Mobility Modular Component Designs 

• Mobility Modular Component Development 

• Modular Robotics Testbed Development 

• Modular Prototype Testing on the Testbed 

Modular, reconfigurable manipulator arm 
and mobility subsystems as part of modular, 
reconfigurable intelligent robots will reduce cost, 
reduce development time and cost, enable more 
uses through reuse and reconfiguration, reduce 
maintenance and repair time and costs, and 
increase availability (uptime). This approach also 
enables low cost, rapid prototyping, and rapid 
development of intelligent robots with testing 
against intended tasks and environments to 
improve quality. All applications are beneficial, 
especially those for space. 


Integrated Sensing and Perception Capabilities for 
Robotics 

Development of the capability to select and 
tailor sensors and real-time perception processing 
to the task- and environment-driven requirements 
of adaptive robots is the objective of this portion 
of the effort. Perception is the extraction of useful 
information about the environment needed to 
understand the situation to complete the task 
successfully. 


These capabilities must be integrated with 
the interface standards of a generic, supervised 
intelligent robotics software architecture. These 
are the most important capabilities enabling 
reactive behavior and deliberative, goal-achieving 
planning and actions. By enabling advanced 
robots to recognize their dynamic environments so 
as to respond appropriately, this effort leads to 
improved productivity by the end users of these 
robots, a growing robotics industry that produces 
jobs and profits, and improved global competitive- 
ness. In space, these capabilities enable robots to 
provide the flexible support that enables space 
exploration (by human/robot teams). 

Integration of sensing and perception into 
planning and control in a robust way is a challenge 
for at least two fundamental reasons. First, the 
time available to sense and perceive the many 
dynamic and unpredictable elements of the 
situation is limited. Second, perception attaches 
meaning to the link between a conception of the 
environment and the objective environment. 
Perception is the process of inference that recog- 
nizes regularities in sensor data that are known on 
the basis of a model of the world to be reliably 
related to causal structure of objects and their 
relations in the environment and then conveys this 
to cognition. Sensory data underdetermines world 
structure; therefore, a model of world structure is 
required. 

Perception involves understanding generic, 
generally applicable models of world structure 
(not merely specific object models) and how that 
causal structure evidences itself in sensor data. 
Causal structure is of interest so as to be able to 
predict consequences, anticipate events, and plan 
actions so as to achieve goals. Perception is 
generally focussed by needs for information that 
supports planning and reasoning for goals 
achievement. Designing perception involves 
converting the understanding and inference 
processes into calculational steps (algorithms and 
inferences) and designing computation hardware 
systems to meet the requirements of information 
at rates and latencies required to deal with a 
dynamic environment. 

The proposed effort consists of partnerships 
between manufacturers, nonprofits, universities, 
and JSC to develop a set of sensors and perception 
processing appropriate to numerous task- and 
environment-driven requirements for adaptive 
robot applications, both in the U.S. economy and 
in space. Included here are vision sensing and 
visual perception, along with speech recognition 
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and task limited natural language understanding 
(speech perception). The unification of visual and 
speech perception is also included here. Proximity 
sensing, tactile/slip sensing, and force/torque 
sensing, which are critical aspects of many manipu- 
lation tasks, are addressed in the next category of 
tasks. 

Current sensing and perception efforts at JSC 
include focused developments for EVA helper/ 
retriever (laser scanner, stereo video, torque and 
proximity sensors, speech recognition and task 
limited, natural language understanding, etc.) and 
the mobile robotics testbed project (real-time 
stereo vision). 

The six tasks in this category are the 
following: 

• Vision Sensors and Sensing Software 
Development 

• Finding, Recognizing, Locating, and Tracking 
Objects and Humans 

• Visual Perception of Objects' Spatial Relations 

• Visual Perception of Objects' Condition and 
Process Participation 

• Speech Recognition and Natural Language 
Understanding 

• Unification of Visual and Speech Perception 

The benefit of sensing and perception 
capabilities is to enable the supervised intelligent 
robot both to extract needed information about 
the changing task environment, including humans, 
on a real-time basis so as to react safely and 
appropriately, and to build and continuously 
update internal representations of the changing 
environments so as to plan safe goal-achieving 
actions. People will have a natural means of 
supervision through task limited, natural language 
understanding software. The unification of visual 
and speech perception adds power to the human/ 
robot team. These benefits apply in space and in 
the U.S. terrestrial economy. 


Robotic Surrogates for Human Grasping and 
Manipulation 

Robots and humans must be capable of 
interacting with the same environment in terms of 
grasping and manipulation for certain tasks. 
Dexterous robotic grasping and manipulation 
capability must be developed to achieve this 
capability. The robot may operate in conjunction 
with a human as an apprentice or may be substi- 


tuted for a human (e.g., in hazardous operations). 
The benefits to the U.S. economy from robots with 
such capability would be very large: improved 
global competitiveness; improved productivity by 
the end users; a growing robotics industry mean- 
ing more jobs and profits; and an increased 
standard of living in the United States. In space, 
robots with these capabilities are required to 
interface with space hardware on astronaut/robot 
teams. This would reduce the cost of designing the 
robotic environment and allow more tasks to be 
done robotically. 

The proposed effort consists of partnerships 
between manufacturers and JSC to develop 
working production prototypes of human-scale 
versions of robot hands by leveraging current 
development by both sides. Integration of tactile, 
slip, force, and torque sensing; adaptive grasping; 
stable grasp recognition; and manipulation strat- 
egy approaches will be accomplished. However.it 
should be recognized that the resulting robot 
hands are not expected to be equivalent to human 
hands. Limited multitask capability is all that is 
expected in the 5-year term of this effort. 

EVA helper/retriever and the dexterous, 
anthropomorphic robotic testbed (DART) are two 
of the current related efforts at JSC, as well as some 
SBIR developments. 

The nine tasks in this category are the 
following: 

• Hand designs 

• Integrated hand, wrist, and arm designs 

• Tactile/slip sensors, sensing software, and 
perception software 

• Proximity sensors, sensing software, and 
perception software 

• Force/torque sensors, sensing software, and 
perception software 

• Integrated sensing with hand, wrist, arm to 
provide stable grasp recognition and other 
intelligent functions 

• Grasping and manipulation strategies 

• Collision avoidance strategies 

• Compliance strategies 

Supervised intelligent robots and human 
ability to interact with the same environment in 
terms of dexterous grasping and manipulation will 
provide major benefits in U.S. industry, service 
applications, and in space. Costly special designs 
and structuring of the robot environment will be 
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minimized or eliminated, thus reducing costs. 
Robots will be able to operate in conjunction with 
humans as robot apprentices to humans on 
human/robot teams. 


An Integrated Prototyping Environment for 
Robotics 

The objective is to develop an integrated 
rapid prototyping and rapid development 
environment for building robotic systems "faster, 
better, and cheaper" based on modularity, 
reconfigurability, and extendibility, including a 
library of hardware modules (such as manipulators, 
tools, and sensors), complementary software 
modules (such as sensing and perception strategies 
and manipulator control), and software advisors 
designed to reduce the cost of programming 
robots. This effort would also leverage develop- 
ment of a generic intelligent robotics software 
architecture in a related subcategory. 

The proposed effort consists of partnerships 
between manufacturers and JSC to develop an 
integrated prototyping environment that will 
allow users to generate and evaluate alternate 
approaches to the design of a robotic system 
quickly. This effort would also leverage develop- 
ment of modular manipulation and mobility for 
robots and other related subcategories. 

Task-directed process design with a systems 
engineering focus and reconfigurable modular 
designs are strong points of our experience at JSC 
that are critical to success here. 

The five tasks in this category are the 
following: 

• Requirements for Prototyping Environment 

• Design of Prototyping Environment 

• Development of Prototyping Environment 

• Testing of Prototyping Environment 

• Knowledge Support System 

Automation of the process of designing and 
developing intelligent robots reduces costs and 
development time. Automation of the testing of 
intelligent robots also reduces costs and develop- 
menttime while providing the user early feedback 
that the robots will solve the problems. All 
markets benefit: advanced manufacturing, health 
care, service industries, construction, mining, 
space, etc. 


Robotic Applications in Advanced Manufacturing. 
Health Care. Service Industries, and Construction 

This effort's objectives are to enable the 
manufacture and marketing of supervised 
intelligent robotic systems for applications in 
advanced manufacturing, health care, service 
industries, and construction by developing 
working production prototypes. Production 
prototypes will also be developed for inspection, 
servicing, maintenance, and repair tasks for space 
exploration. Such advanced robotic systems offer 
the benefits of improved productivity by the end 
users and improved global competitiveness to the 
U.S. economy. In space, such robots provide 
improved safety, reliability, and productivity as 
Space Station evolves and enables human space 
exploration (by human/robot teams). 

The proposed effort consists of partnerships 
between manufacturers, nonprofits, doctors and 
hospitals, universities, and JSC. 

The required core skills are available at JSC in 
its civil service and contractors to form the nucleus 
of the multiple partnerships. Current technology 
integration efforts include the EVA helper/ 
retriever supervised intelligent robot, the mobile 
robotics testbed project, and the Soda-Pup entry in 
the AAAI national robotics competition. In 
addition, JSC is responsible for numerous 
applications of robotics 

In our ongoing relationship with the Texas 
Medical Center, recent interest by Drs. Steve Kroll 
and Chuck Van Duren in robotic microsurgery and 
arterial catheterization has been shown. 

The four tasks in this category are the 
following: 

• Robotic Applications in Advanced 
Manufacturing 

• Robotic Applications in Health Care 

• Roboti c Appl i cati ons i n Servi ce I nd ustri es 

• Robotic Applications in Construction 

The benefits of intelligent robots to advanced 
manufacturing are spelled out in detail in 
Erickson 1 . The benefit to health care is lower cost 
health care delivery with quality improvements 
due to improvements in productivity. The benefits 
to other service industry applications are improve- 
ments in productivity. The benefits of intelligent 
robots to construction include improved construc- 
tion time and productivity. 
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3. Concluding Remarks 

We have presented a "straw man" pre- 
competitive, dual use technology program in 
intelligent robotics intended to stimulate the 
creative aspects of nonaerospace and aerospace 
industry to propose their own objectives, 
approaches, and tasks for new jointly funded 
partnerships with NASA JSC. It is evidence of our 
"earnest" and that we are ready to proceed with 
our end of the partnerships. 
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